How to Gauge Accuracy of Measurements and of Expert Estimates: Beyond Normal Distributions

To properly process data, we need to know the accuracy of different data points, i.e., accuracy of different measurement results and expert estimates. Often, this accuracy is not given. For such situations, we describe how this accuracy can be estimated based on the available data. 1 Formulation of the Problem Need to gauge accuracy. To properly process data, it is important to know the accuracy of different data values, i.e., the accuracy of different measurement results and expert estimates; see, e.g., [3–5]. In many cases, this accuracy information is available, but in many other practical situations, we do not have this information. In such situations, it is necessary to extract this accuracy information from the data itself. Extracting uncertainty from data: traditional approach. The usual way to gauge of the uncertainty of a measuring instrument is to compare the result x̃ produced by this measuring instruments with the result x̃s of measuring the same quantity x by a much more accurate (“standard”) measuring instrument. Since the “standard” measuring instrument is much more accurate than the instrument that we are trying to calibrate, we can safely ignore the inaccuracy of its measurements and take x̃s as a good approximation to the actual value x. In this case, the difference x̃− x̃s between the measurement results can serve as a good approximation to the desired measurement accuracy ∆x = x̃− x. Traditional approach cannot be applied for calibrating state-of-the-art measuring instruments. The above traditional approach works well for many measuring instruments. However, we cannot apply this approach for calibrating state-ofthe-art instrument, because these instruments are the best we have. There are no other instruments which are much more accurate than these ones – and which can therefore serve as standard measuring instruments for our calibration. Such situations are ubiquitous; for example: – in the environmental sciences, we want to gauge the accuracy with which the Eddy covariance tower measure the Carbon and heat fluxes; see, e.g., [1]; – in the geosciences, we want to gauge how accurately seismic [2], gravity, and other techniques reconstruct the density at different depths and different locations. How state-of-the-art measuring instruments are calibrated: case of normally distributed measurement errors. Calibration of state-of-the-art measuring instruments is possible if we make a usual assumption that the measurement errors are normally distributed with mean 0. Under this assumption, to fully describe the distribution of the measurement errors, it is sufficient to estimate the standard deviation σ of this distribution. There are two possible approaches for estimating this standard deviation. The first approach is applicable when we have several similar measuring instruments. For example, we can have two nearby towers, or we can bring additional sensors to the existing tower. In such a situation, instead of a single measurement result x̃, we have two different results x̃ and x̃ of measuring the same quantity x. Here, by definition of the measurement error, x̃ = x + ∆x and x̃ = x+∆x and therefore, x̃ − x̃ = ∆x −∆x. Each of the random variables ∆x and ∆x is normally distributed with mean 0 and (unknown) standard deviation σ (i.e., variance σ). Since the two measuring instruments are independence, the corresponding random variables ∆x and ∆x are also independent, and so, the variance of their difference is equal to the sum of their variances σ +σ = 2σ. Thus, the standard deviation σ′ of this difference is equal to √ 2 · σ. We can estimate this standard deviation σ′ based on the observed differences x̃ − x̃ and therefore, we can estimate